The present invention relates to digital watermarking of video signals, and more particularly to temporal synchronization of video watermark decoding with video watermark encoding via an embedded synchronization pattern.
Digital watermarking of video signals by embedding invisible patterns into the active area of the video signals as images or digital data representations has been proposed as a viable solution to the need for copyright protection and authentication of the video signals, as discussed in U.S. Pat. No. 4,969,041. Similarly audio program material may be watermarked as well. Other applications have been proposed for the data payload of a watermarked video signal, as described in U.S. Pat. No. 6,211,919.
Since imperceptibility is important, various methods have been proposed for video and audio watermarking using masking characteristics of the Human Visual/Audio System. In the case of video watermarking a model of the Human Visual System is used to sample the video signal, often both spatially and temporally, in order to maximize the magnitude of the watermark pattern in the video signal while maintaining an acceptably low or imperceptible visibility. This is shown in FIG. 1 where a video signal is input to a Human Visual System mask generator and a watermark adder. The mask from the mask generator is used as a gain control signal to multiply on a pixel by pixel basis watermark noise-like patterns, which in turn are added to the video signal by the watermark adder to produce the watermarked video signal for distribution. The distributed watermarked video signal is received at a decoder and filtered, using a suitable filter such as a video rejection or other whitening filter, the filtered signal being correlated with the watermark noise-like patterns to determine the presence of one or more of the watermark noise-like patterns in the video signal and accordingly the data or image represented by the detected watermark noise-like patterns.
A second requirement for video watermarking is robust detection of the embedded watermark pattern after image distortions, such as translation of the image within the video frame, image cropping, geometric expansion/contraction/rotation, additive noise and image distortion due to MPEG or other lossy compression methods. Video compression and resynchronization may cause repeated or missing frames in the decompressed video output requiring the embedded watermark to be error corrected over a block of frames where not all of the watermark encoded frames are received. Correcting data is made easier if the decoder synchronizes to the same block of frames that the encoder encodes in order to detect and correct for data errors due to such repeated or dropped video frames. In fact periodic block synchronization may be necessary if the video sequence has been shortened by dropping frames for other reasons, such as commercial insertion.
Often a third requirement is the ability to detect that a video sequence has been previously watermarked so the encoder either stops the watermarking process or overwrites the watermark with a new watermark pattern orthogonal to the already existing watermark. In the latter situation the hierarchical watermarking allows a decoder to ignore the previous or lower hierarchical watermark and to decode the later or higher hierarchical watermark.
Some video watermarking methods, as shown in FIG. 2, process the video signal through a spatial or spatial-temporal image transform T before embedding the watermark patterns in the image, such as by altering the transform coefficients. The patterns may be full field images or may be subdivided into identical image tiles. An inverse transform is performed to provide the watermarked video signal for distribution. At the decoder the same image transform is performed on the watermarked video signal, and the resulting transformed signal is correlated with the original watermark pattern or tile to provide the data represented by the watermark pattern. These transform methods are useful for both improving the imperceptibility and the robustness to image distortion of the embedded watermark pattern, but at the expense of increased computational complexity in performing the image transform and determining image distortion.
Where the watermark data payload is spread over a block of frames, a watermark data decoder needs to be synchronized with the watermark encoder over the same block of frames. This temporal synchronization is advantageous for audio to video timing measurement or correction and for timing realtime metadata related to the video signal, among other applications. One obvious method of synchronizing the decoding over the same block of frames as those encoded is by using a data header in the encoded data stream to mark the boundaries of each block. However this data may be corrupted causing the block to be missed, and the data payload (number of information data bits encoded) is reduced.
What is desired is a method of temporally synchronizing a watermark video decoder with a corresponding watermark encoder over a block of frames in order to robustly determine when the watermark is present as well as the watermark signal level, to have a computationally simple method of tracking small image shifts, to recognize a previously watermarked sequence and add a higher priority level watermark pattern so that the decoder may detect either watermark pattern without error.
Accordingly the present invention provides temporal synchronization of video watermark decoding with a watermark encoder via an embedded three-dimensional synchronization pattern encoded into a watermarked video signal over a block of frames without adding data overhead. At a watermark encoder a spatial (2D) synchronization vector multiplied by a pseudo-noise (PN) sequence of apparently random plus and minus ones that determines the polarity of the spatial synchronization vector in each frame over the block of frames to produce a spatio-temporal (3D) synchronization pattern. The 3D synchronization pattern is added to an information data pattern, modulated and further processed before finally being added to a video signal as a watermark pattern to produce the watermarked video signal. At a watermark decoder after preprocessing and demodulating the watermarked video signal is correlated with the data pattern and with the spatial synchronization vector. The correlated spatial synchronization vector is then input to a PN synchronizer that detects a temporal offset between the watermark decoder and the watermark encoder. The temporal offset is used to load a frame counter to produce frame block counts in temporal synchronization with the watermark encoder. The synchronization vector and data pattern may have I and Q components which are modulated by a quadrature carrier signal before being added together as the watermark pattern. Once video frame block synchronization is achieved, forward error correction over the block of video frames is possible using Reed-Solomon codes. Also to maintain synchronization when frames are dropped or added, the block of video frames may be subdivided into sub-blocks with synchronization correlation being done at the sub-block level to detect a missing frame or added frame in the sub-block and then padding the sub-block or dropping frames from the sub-block to maintain synchronization over the block of video frames. Also by modulating an addition data pattern with Walsh codes an additional robust, low data rate channel may be added. With a combination of hard-decisions at high data rates and soft-decisions at low data rates a gradual degradation of data detection rather than a xe2x80x9ccliff-effectxe2x80x9d degradation may be achieved. Also the temporal correlation of the spatial synchronization vectors may be used to generate a signal level indication for the watermarked data to determine a loss of signal condition. Further hierarchical watermarking is enabled by using orthogonal spatial synchronization vectors with separate sets of data vectors so that the watermark decoder effectively decodes the watermark with the higher hierarchy synchronization vector, effectively erasing the lower hierarchy watermark. Finally shifts of the active picture containing watermarks may be detected and corrected using the synchronization pattern.
The objects, advantages and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing.